Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)

Deep neural networks (DNN) have shown remarkable success in a variety of machine learning applications. The capacity of these models (i.e., number of parameters), endows them with expressive power and allows them to reach the desired performance. In recent years, there is an increasing interest in deploying DNNs to resource-constrained devices (i.e., mobile devices) with limited energy, memory, and computational budget. To address this problem, we propose Entropy-Constrained Trained Ternarization (EC2T), a general framework to create sparse and ternary neural networks which are efficient in terms of storage (e.g., at most two binary-masks and two full-precision values are required to save a weight matrix) and computation (e.g., MAC operations are reduced to a few accumulations plus two multiplications). This approach consists of two steps. First, a super-network is created by scaling the dimensions of a pre-trained model (i.e., its width and depth). Subsequently, this super-network is simultaneously pruned (using an entropy constraint) and quantized (that is, ternary values are assigned layer-wise) in a training process, resulting in a sparse and ternary network representation. We validate the proposed approach in CIFAR-10, CIFAR-100, and ImageNet datasets, showing its effectiveness in image classification tasks.Comment: Proceedings of the CVPR'20 Joint Workshop on Efficient Deep Learning in Computer Vision. Code is available at https://github.com/d-becking/efficientCNN

Secure Clamping of Parts for Disassembly for Remanufacturing

Robot based remanufacturing of valuable products is commonly perceived as promising field in future in terms of an efficient and globally competitive economy. Additionally, it plays an important role with regard to resource-efficient manufacturing. The associated processes however, require a reliable non-destructive disassembly. For these disassembly processes, there is special robot periphery essential to enable the tasks physically. Unlike manufacturing, within remanufacturing there are End-of-Life (EoL) products utilized. The specifications and conditions are often uncertain and varying. Consequently the robot system and especially the periphery needs to adapt to the used product, based on an initial examination and classification of the part. State of the art approaches provide limited flexibility and adaptability to the disassembly of electric motors used in automotive industry. Especially the geometrical shape is a limiting factor for using state of the art periphery for remanufacturing. Within this contribution a new kind of flexible clamping device for the disassembly of EoL electrical motors is presented. The robot periphery is systematically developed regarding the requirements stemming from the remanufacturing approach. It consists of three clamping units with moveable pins. Utilizing two linear axes, a two dimensional working space is realized for clamping the parts depending on their conditions and shape

Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication

Currently, progressively larger deep neural networks are trained on ever growing data corpora. As this trend is only going to increase in the future, distributed training schemes are becoming increasingly relevant. A major issue in distributed training is the limited communication bandwidth between contributing nodes or prohibitive communication cost in general. These challenges become even more pressing, as the number of computation nodes increases. To counteract this development we propose sparse binary compression (SBC), a compression framework that allows for a drastic reduction of communication cost for distributed training. SBC combines existing techniques of communication delay and gradient sparsification with a novel binarization method and optimal weight update encoding to push compression gains to new limits. By doing so, our method also allows us to smoothly trade-off gradient sparsity and temporal sparsity to adapt to the requirements of the learning task. Our experiments show, that SBC can reduce the upstream communication on a variety of convolutional and recurrent neural network architectures by more than four orders of magnitude without significantly harming the convergence speed in terms of forward-backward passes. For instance, we can train ResNet50 on ImageNet in the same number of iterations to the baseline accuracy, using $\times 3531$ less bits or train it to a $1\%$ lower accuracy using $\times 37208$ less bits. In the latter case, the total upstream communication required is cut from 125 terabytes to 3.35 gigabytes for every participating client

Bond activation in iron(II) and nickel(II) complexes of polypodal phosphanes

A pyridine-derived tetraphosphane ligand (donor set: NP4) has been found to undergo remarkably specific C-P bond cleavage reactions, thereby producing a ligand with an NP3 donor set. The reaction may be reversed under suitable conditions, with regeneration of the original NP4 ligand. In order to investigate the mechanism of this reaction, the NP3 donor ligand C5H3N[CMe(CH2PMe2)2][CMe2(CH2PMe2)] (11) was prepd., and its iron(II) complex 4 generated from Fe(BF4)2·6 H2O, with Me diethylphosphinite (7) as an addnl. monodentate ligand. Ligand 11 has, in addn. to the NP3 donor set, one Me group in close contact with the iron center, reminiscent of an agostic M···H-C interaction. Depending on the stoichiometric amt. of iron(II) salt, a side product 15 is formed, which has a diethylphosphane ligand instead of the phosphinite 7 coordinated to iron(II). While attempts to deprotonate the metal-coordinated Me group in 4 were unsuccessful, the reaction was shown to occur in an alternative complex (18), which is similar to 4 but has a trimethylphosphane ligand instead of the phosphinite 7. The reaction of complex 15 with CO gave two different products, which were both characterized by single-crystal X-ray diffraction. One (19) is the dicarbonyl iron(II) complex of the triphosphane ligand 11, the other (3) is the carbonyl iron(II) complex of the tetraphosphane C5H3N[CMe(CH2PMe2)2]2 (1). This suggests an intermol. mechanism for the C-P bond formation in question. [on SciFinder(R)

Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning

The success of convolutional neural networks (CNNs) in various applications is accompanied by a significant increase in computation and parameter storage costs. Recent efforts to reduce these overheads involve pruning and compressing the weights of various layers while at the same time aiming to not sacrifice performance. In this paper, we propose a novel criterion for CNN pruning inspired by neural network interpretability: The most relevant units, i.e. weights or filters, are automatically found using their relevance scores obtained from concepts of explainable AI (XAI). By exploring this idea, we connect the lines of interpretability and model compression research. We show that our proposed method can efficiently prune CNN models in transfer-learning setups in which networks pre-trained on large corpora are adapted to specialized tasks. The method is evaluated on a broad range of computer vision datasets. Notably, our novel criterion is not only competitive or better compared to state-of-the-art pruning criteria when successive retraining is performed, but clearly outperforms these previous criteria in the resource-constrained application scenario in which the data of the task to be transferred to is very scarce and one chooses to refrain from fine-tuning. Our method is able to compress the model iteratively while maintaining or even improving accuracy. At the same time, it has a computational cost in the order of gradient computation and is comparatively simple to apply without the need for tuning hyperparameters for pruning.Comment: 25 pages + 5 supplementary pages, 13 figures, 6 table